Managing remote data replication

ABSTRACT

Various systems, processes, and products may be used to manage remote data replication. In particular implementations, a system and process for managing remote data replication may include the ability to store versions of a disk at a first site, a second site, and a third site. The version of the disk at the first site may store input/output for a host system, the version at the second site may be a synchronous replication of the version at the first site, and the version at the third site may be an asynchronous replication of the version at the first site. The system and process may also include the ability to synchronize the version at the first site with the version at the third site if the second site is unavailable and synchronize the version at the second site with the version at the third site if the first site is unavailable.

BACKGROUND

The present invention relates to managing computer data, and morespecifically to managing data replication.

It is common for a storage system to have one or more of its disksbacked up at a remote location to allow for disaster recovery. Moststorage systems support simple point-to-point remote replication, fromone source disk to one target disk, although some enterprise systems dosupport three-site replication. The replications may be performed in asynchronous or asynchronous manner.

In synchronous replication, the source storage system does not reportcompletion of an operation (e.g., a write) until the target storagesystem has completed the operation. Thus, while guaranteeing anidentical copy of a disk, this process is relatively slow and typicallyforces the target disk to be located relatively near the source disk.

In asynchronous replication, the source storage system can reportcompletion of an operation before the target storage system hascompleted the operation (e.g., upon receiving notice that the targetstorage system has received the communication conveying the operation).Thus, this process is relatively fast and allows the target disk to belocated farther away from the source disk, but does not guarantee anidentical copy of the disk.

BRIEF SUMMARY

In one implementation, a process for managing remote data replicationmay include storing a version of a disk at a first site, storing aversion of the disk at a second site that is remote from the first site,and storing a version of the disk at a third site that is remote fromthe first site and the second site. The version of the disk at the firstsite may, for example, be adapted for storing input/output for a hostsystem, the version of the disk at the second site may be a synchronousreplication of the version at the first site, and the version of thedisk at the third site may be an asynchronous replication of the versionat the first site. The process may also include synchronizing theversion of the disk at the first site with the version at the third siteif the version at the first site needs synchronizing and the version atthe second site is unavailable and synchronizing the version of the diskat the second site with the version at the third site if the version atthe second site needs synchronizing and the version at the first site isunavailable.

The details and features of various implementations will be conveyed bythe following description, along with the drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example system for managingremote replication of data.

FIG. 2 is a block diagram illustrating an example configuration of asystem for managing remote replication of data.

FIG. 3 is a flowchart illustrating an example process for managingremote replication of data.

FIGS. 4A-D are a flowchart illustrating another example process formanaging remote replication of data.

FIGS. 5A-B are a flowchart illustrating an additional example processfor managing remote replication of data.

FIG. 6 is a flowchart illustrating a further example process formanaging remote replication of data.

FIG. 7 is a block diagram illustrating an example computer system formanaging remote replication of data.

DETAILED DESCRIPTION

Managing of remote data replication may be achieved by varioustechniques. In particular implementations, the management of remote datareplication may include providing three sites, two of which aresynchronously replicated to each other and one of which isasynchronously replicated from the first. Upon losing any one site, theother sites may continue to perform data storage functions. Moreover,the site that was lost may be resynchronized from either of the stilloperational sites. Thus, a robust system is provided.

As will be appreciated by one skilled in the art, aspects of the presentdisclosure may be implemented as a system, method, or computer programproduct. Accordingly, aspects of the present disclosure may take theform of an entirely hardware environment, an entirely softwareembodiment (including firmware, resident software, micro-code, etc.), oran implementation combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module,” or “system.”Furthermore, aspects of the present disclosure may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of a computer readable storagemedium would include the following: an electrical connection having oneor more wires, a portable computer diskette, a hard disk, a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), an optical fiber, a portablecompact disc read-only memory (CD-ROM), an optical storage device, amagnetic storage device, or any suitable combination of the foregoing.In the context of this disclosure, a computer readable storage mediummay be a tangible medium that can contain or store a program for use byor in connection with an instruction execution system, apparatus, ordevice.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any medium, including but not limited to wireless, wireline,optical fiber cable, RF, etc. or any suitable combination of theforegoing.

Computer program code for carrying out operations for aspects of thedisclosure may be written in any combination of one or more programminglanguages such as Java, Smalltalk, C++ or the like and conventionalprocedural programming languages, such as the “C” programming languageor similar programming languages. The program code may execute entirelyon the user's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer, or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the disclosure are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to implementations.It will be understood that each block of the flowchart illustrationsand/or block diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor of a general purpose computer, special purposecomputer, or other programmable data processing apparatus to produce amachine, such that the instructions, which execute via the processor ofthe computer or other programmable data processing apparatus, createmeans for implementing the functions/acts specified in the flowchartand/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other device to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions thatimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus, or other devices to produce a computerimplemented process such that the instructions that execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

FIG. 1 illustrates an example system 100 for managing the remotereplication of data. System 100 includes storage systems 110, acommunication network 120, and a communication network 130.

Storage systems 110 may store data for various host systems (e.g.,server systems running Advanced Interactive eXecutive (AIX) or Windows)that may be local or remote from the storage systems. In particularimplementations, storage system 110 a may provide the primary storage,and storage systems 110 b-c may provide the backup storage. Storagesystems 110 may, for example, include storage area networks, raidarrays, hard drives, tape drives, or any other device for storing data.Storage systems 110 may, for instance, store data in a block array, filesystem, database, or other appropriate format. Each of storage systems110 includes a storage controller 112 and storage 114 (e.g., a harddisk, a tape, etc.).

Storage controllers 112 are responsible for actually placing data instorage 114. Additionally, storage controllers 112 coordinate with eachother so that an operation (e.g., a write) at one of storage systems 110is applied to the other storage systems 110. Each storage 114 includes areplica of a disk 115, which is an item that the storage systems aretrying to be keep consistent between them. Disk 115 may be an actualdisk in storage 114 or a representation of a disk in storage 114 (e.g. avirtualization).

A replication process basically keeps two disks consistent by updating asecond disk with changes at a first disk as they occur. Replicationsbetween the disks may be in a synchronous or an asynchronous manner. Asynchronous replication involves the second disk being updated in acontemporaneous manner as changes to the first disk occur. Thus, in mostcases, the same data would be read from each disk at the same point intime. An asynchronous replication also involves the second disk beingupdated as changes to the first disk occur but not always in acontemporaneous manner. Thus, the data on the disks may be different fora period of time.

Note that although the techniques are described herein as beingapplicable to a single series of disks 115, storage systems 110 may havea number of disks that are being replicated between them, and thedescribed techniques may be applied to any series of disks. For example,if storage system 110 a has 100 disks being replicated to 100 disks ofstorage system 110 b and storage system 110 c, the following design canbe applied to each set of disks individually.

Storage systems 110 a-b are communicatively coupled to each otherthrough communication network 120. Communication network 120 may, forexample, be a local area network, a wide area network, or the Internet.Storage systems 110 a-b are coupled to communication network 120 throughcommunication links 122, which may include one or more wireline (e.g.,cable or fiber optic) and/or wireless links (e.g., radio frequency).

Storage systems 110 a-b are communicatively coupled to storage system110 c through communication network 130. Communication network 130 may,for example, be a local area network, a wide area network, or theInternet. Storage systems 110 a-c are coupled to communication network130 through communication links 132, which may include one or morewireline (e.g., cable or fiber optic) and/or wireless links (e.g., radiofrequency).

In certain modes or operation, storage systems 110 a-b operate in asynchronous manner for disk 115. Thus, when storage system 110 a hasdata to write to disk 115, the host system will not receive successfulcompletion until storage system 110 b has committed the write.Additionally, storage system 110 a-b and storage system 110 c operate inan asynchronous manner for disk 115. That is, storage systems 110 a-bmay count a write as complete without having to wait for storage system110 c to commit the write.

As discussed previously, disks 115 are each on a separate storagesystems 110. Each system may, for example, be running similar code onsimilar hardware. Disk 115 a is the master copy to be replicated todisks 115 b and 115 c, so during normal operations, disk 115 a isreceiving writes from the host system. Disk 115 b is typically at arelatively short distance from disk 115 a. For example, disk 115 b maybe on the same campus or within about 100 km. As indicated above, thereplication between disk 115 a and disk 115 b is synchronous, so thereplication of disk 115 a at disk 115 b has a recovery point objective(RPO) of zero. For example, disk 115 a and disk 115 b can provide anenvironment suitable for automated storage failover such as thatprovided by GDPS™ HyperSwap from International Business MachinesCorporation (IBM) of Armonk, N.Y. or GDDR AutoSwap from EMC Corporation(EMC) of Hopkinton, Mass.

Disk 115 c is typically at a relatively large distance from both disk115 a and disk 115 b. Thus, disk 115 c may provide out-of-regiondisaster recovery. For example, disk 115 c may be located at a distancein excess of 100 km from the other disks. As indicated previously, thereplication between disk 115 a and disk 115 c and between disk 115 b anddisk 115 c is asynchronous, so the replication of disk 115 a at disk 115c and disk 115 b at disk 115 c has a non-zero RPO.

In some modes of operation, disk 115 a and disk 115 b are the source ofasynchronous replications to disk 115 c at different times. Inparticular, this may depend on which of disk 115 a and disk 115 b iscurrently available. In some implementations, the source of replicationto disk 115 c may move flexibly between disk 115 a and disk 115 b,depending whether a star or a cascade topology is used.

By sequencing of point-in-time copies and point-to-point replication,which will be discussed in more detail below, system 100 can achieve athree-site replication solution while being able to incrementallyresynchronize between any two disks in the absence of the third. Thisallows a robust, complete solution made from a small number of buildingblocks.

System 100 has a variety of features. For example, data may be readilyreplicated to multiple locations, which may provide increased resilienceor wide distribution of data (e.g., a golden image). In particular,system 100 provides local highly available redundant copies while stillsupporting disaster recovery at out-of-region distances. Many existingstorage systems only support one target per host-accessible disk.

Although FIG. 1 illustrates one implementation of a system for remotedata replication, other systems may include fewer, additional, and/or adifferent arrangement of components. For example, various other systems(e.g., server systems) may be co-located with one or more storagesystems. As another example, communication network 120 and communicationnetwork 130 could be the same communication network (e.g., theInternet). Furthermore, disks 115 could be actual disks (e.g., harddrives).

FIG. 2 illustrates an example system configuration 200 for managingremote data replication. System configuration 200 may, for example, beimplemented by system 100. Note that system configure 200 generallyshows the data flows for backing up disk 212 a. However, the data flowsmay be reversed for various situations (e.g., resynchronizing a sitethat has been lost).

As illustrated, system configuration 200 includes a number of sites 210,each of which has a number of disks 212. Disk 212 a is the main copy,and disks 212 b-g are replicas of disk 212 a, although they may havedifferent versions of disk 212 a at various times. It should beunderstood that disks 212 may not be actual disks, but representationsof disks in storage. Each of sites 210 may be a separate storage systemin a separate location.

As discussed in more detail below, system configuration 200 uses anumber of point-in-time copies and synchronous replications. Some of thepoint-in-time copies are multiple-target, and the system configurationcalls for both replicating a point-in-time copy and taking apoint-in-time copy of a replication target. The system configurationmay, for example, be implemented using a Storage Area Network (SAN)Volume Controller from IBM, which supports these features as of V6.2.

In certain modes of operation, assuming only disk 212 a is in existence,disks 212 b-g are then created (operation A1). Then, synchronousreplications may be created but not started between disk 212 a and disk212 e, disk 212 b and disk 212 c, and disk 212 f and disk 212 g(operation A2). Synchronous disk replications may, for example, beaccomplished with Metro Mirror from IBM, Symmetrix Remote Data Facility(SRDF) from EMC Corporation, or TrueCopy from Hitachi Data SystemsCorporation (HDS) of Santa Clara, Calif.

Additionally, point-in-time copies may be created but not startedbetween disk 212 a and disk 212 b, disk 212 e and disk 212 f, disk 212 cand disk 212 g, and disk 212 c and disk 212 d (operation A3).Point-in-time copies may, for example, be accomplished with FlashCopyfrom IBM, TimeFinder/Clone from EMC, or ShadowImage from HDS. Thepoint-in-time copies may be configured to perform no background copy.

Next, the synchronous replication between disk 212 a and disk 212 e isstarted (operation A4). This process may generally run throughoutoperations of system configuration 200. The process may, for example, bepaused when on of disk 212 a or disk 212 e is unavailable.

After the synchronous replication between disk 212 a and disk 212 e hasfinished its initial copy (operation A5), the point-in-time copies ofdisk 212 a to disk 212 b and disk 212 e to disk 212 f may be triggeredwith respect to host system input/output (TO) (operation A6). This mayinvolve quiescing host system IO. Disk 212 b may, for example, be alogical copy of disk 212 a. Then, the synchronous replication betweendisk 212 b and disk 212 c may be started (operation A7).

Once the synchronous replication between disk 212 b and disk 212 c hasfinished its initial copy, the replication is stopped (operation A8).Next, the point-in-time copies from disk 212 c to disk 212 d and fromdisk 212 c to disk 212 g is triggered (operation A9). The relativetiming of these copies is typically unimportant. After this, consistentimages exist on disk 212 a, disk 212 e, and disk 212 d, although disk212 d will probably have an older image than disk 212 a and disk 212 e.In certain implementations, disk 212 d may have historical images ofdisk 212 a.

In particular modes of operation, disk 212 d may be the master disk ifdisk 212 a and disk 212 e are unavailable. The failover between disk 212a or disk 212 e to disk 212 d is similar to the failover between disk212 a and disk 212 e, except that that the failover is more involved asall data since the start of the previous cycle will be lost by failingover to disk 212 d. Thus, this transition is not transparent. This is acommon problem with asynchronous replication solutions and is typicallymanaged by the host applications using such storage systems.

Then, the replication between disk 212 f and disk 212 g, which is notcurrently running, may be cleaned (operation A10). Replications mayexist but not run. A replication that is running (e.g., has beenstarted) will attempt to replicate the data according to the configuredrequirements. For example, a synchronous replication may attempt toreplicate data immediately, but an asynchronous replication mayreplicate after a delay. A replication that exists but is not running(e.g., has not been started or has been stopped) will not attempt toreplicate the data, but it typically records which regions have beenchanged, so that when started, it can resynchronize those regions. Usingthis feature of replication technology, the changes to disk 212 f causedby the triggering of the point-in-time copy from disk 212 e are recordedby the replication disk between 212 f and disk 212 g, which dirties thestopped replication. This recording is cumulative, and so for eachcycle, even though the replication between disk 212 f and disk 212 g hasnot started, the replication should be cleaned.

Cleaning the replication between disk 212 f and disk 212 g may, forexample, be accomplished by deleting the synchronous replication andrecreating it in a clean state (i.e. one where if the replication werestarted at that point, it would not attempt to resynchronize disk 212 fand disk 212 g with data that has already replicated through disk 212 ato disk 212 b, disk 212 b to disk 212 c, and disk 212 c to disk 212 g).This is commonly referred to as a NOCOPY creation, and is supported byvarious products, such as DS8000 from IBM and XP Continuous Access fromHewlett-Packard Company of Palo Alto, Calif. Other techniques may alsobe used to clean the replication. For instance, it may be possible todirectly instruct the system to discard all differences recorded betweendisk 212 f and disk 212 g.

After this, the point-in-time copies from disk 212 a to disk 212 b andfrom disk 212 e to disk 212 f may be retriggered (operation A11). Thisdirties the replications from disk 212 b to disk 212 c and from disk 212f to disk 212 g on the regions that have been updated by retriggeringthe point-in-time copies. The process may then be repeated beginning atoperation A7.

The cycle from operation A7 to operation A12 can run as fast as thesynchronous replication can complete copying the changes and the othercommands can be run. As the length of time that the replication copyingtakes is proportional to the number of writes written to disk 212 a inthe previous cycle, while the rate of writes written is less than thereplication copying rate, the cycle period (and hence the RPO) can tendtowards zero. And while the rate of writes is greater than thereplication copying rate, the cycle period and the RPO will grow. Thisallows the system to cope with significant spikes in write rates.

If site 210 a is lost, system configuration 200 can fail host system IOover to site 210 b using the disk 212 a to disk 212 e replication. Thus,host system IO is now being written to disk 212 e. The fail overdetermination may, for example, be accomplished by a high-availabilitysolution such as Hyperswap or PowerHA from IBM. The high-availabilitysolution may be co-located with the storage systems or the host systems.

High-availability solutions tend to be configured to play a relativelypassive role, where they observe the loss of a site and reconfigure thehost systems to access another replica directly or drive a host-levelfailover to a host system that can access another replica (the detailsof which would be specific to the host type). High-availabilitysolutions would typically have the connectivity and knowledge ofavailability needed for performing the sequencing actions (e.g.,synchronizations and point-in-time copies) discussed previously.However, the sequencing actions need not be co-located with ahigh-availability solution.

Except if the process has been performing operations A7-A10, the processcontinues with triggering the point-in-time copy from disk 212 e to disk212 f (operation B1), which is similar to operations A6/A11. The processalso calls for starting the synchronous replication from disk 212 f todisk 212 g (operation B2), which is similar to operation A7, and thenwaiting for the synchronous replication from disk 212 f to disk 212 g tofinish its copy of changed data and stopping it (operation B3), which issimilar to operation A8. After this, a point-in-time copy from disk 212g to disk 212 d may be triggered (operation B4), which is similar tooperation A9. The cycle of operations B1-B4 may be repeated until site210 a returns.

If the process had been performing operation A7 or operation A8 whensite 1 was lost, the process calls for starting at operation B2, as disk212 f already contains the data needed to copy for this cycle. If site 1was lost during operation A9, the process calls for performing operationA10 (i.e., cleaning the F to G synchronization) and starting fromoperation B4. If site 1 was lost during operation A10, the process callsfor performing operation A10, then starting from operation B1.

A difference recording in the stopped replication from disk 212 e todisk 212 a while the process cycles through operations B1-B3 may allowsite 210 a to be incrementally resynchronized when it returns. As disk212 c remains static while the B1-B3 operation cycle occurs, thedifferences recorded by the point-in-time copy from disk 212 c to disk212 d allows incremental resynchronization of site 210 a from site 210 cin the absence of site 210 b by first reversing the direction andstarting the point-in-time copy between disk 212 c and disk 212 d, suchthat disk 212 c is modified to match disk 212 d. The changes this makesto disk 212 c may be recorded in the difference recording of thesynchronous replication between disk 212 b and disk 212 c, such thatthat replication can be restarted from disk 212 c to disk 212 b and willcopy only the necessary regions to resynchronize site 210 a. Reversingthe direction and starting the point-in-time copy between disk 212 a anddisk 212 b to copy the changes on disk 212 b to disk 212 a will completethe restoration of site 210 a.

When site 210 a returns, the synchronous replication between disk 212 eand disk 212 a may be restarted, although in the reverse direction frombefore (operation C1). After the incremental resynchronization of thesynchronous replication from disk 212 e to disk 212 a is complete(operation C2), the process may wait for the current copy from disk 212f to disk 212 g to complete (operation C3). Then, the point-in-time copyfrom disk 212 g to disk 212 d may be triggered (in the reversedirection) (operation C4), and the point-in-time copy from disk 212 g todisk 212 c may be triggered (again in the reverse direction) (operationC5). The disk 212 c to disk 212 d replication may then be cleaned as inoperation A10 (operation C6), and the main sequence may be started fromA11 (operation C7).

At this point, the process may fail back to the host system IO beingreceived on disk 212 a by stopping the disk 212 e to disk 212 areplication and restarting it in the disk 212 a to disk 212 e direction.However, this is not required.

If site 210 b is lost, the host system IO may continue being written todisk 212 a, and the main cycle (i.e., operations A7-A11) may continue,except that nothing on site 210 b may be manipulated, and the disk 212 cto disk 212 g point-in-time copy is not triggered. Thus, disk 212 g maymaintain an older copy of disk 212 c. The synchronous replication fromdisk 212 a to disk 212 e is automatically stopped due to the loss ofsite 2.

The difference recording in the stopped replication from disk 212 a todisk 212 e allows site 210 b to be incrementally resynchronized when itreturns. As disk 212 g remains static, the differences recorded by thepoint-in-time copy from disk 212 g to disk 212 d would also allowincremental resynchronization of site 210 b from site 210 c in theabsence of site 210 a.

When site 210 b returns, the replication from disk 212 a to disk 212 eis restarted. When that replication is complete, and the main cyclecompletes operation A8, the process may continue from operation A9 fullyincluding site 210 b (such as retriggering the disk 210 c to 210 gpoint-in-time copy).

If site 210 c is lost, operations A7-A11 are paused, and the processwaits for site 210 c to return. When it does, the process can continuefrom the operation it was previously at. If site 210 a is not available,however, site 210 c may be resynchronized from site 210 b.

System configuration 200 allows replication to two remote sites thathave differing replication characteristics, with simple and fastresynchronization when losing any one site. System configuration 200also provides the ability to continue working in the absence of any copyby incrementally resynchronizing between the remaining copies.Additionally when the site returns, it may be resynchronized to eitherof the other two sites, so the system configuration can easily cope witha rolling failure that impacts multiple locations at overlapping times.Some enterprise systems support three-site replication, but preventincremental resynchronization between a pair of disks (e.g., disk 212 aand disk 212 d or disk 212 e and disk 212 d), which makes the systemssignificantly less robust when a site is inaccessible.

FIG. 3 illustrates an example process 300 for managing the remotereplication of data. Process 300 may, for example, be performed by asystem similar to system 100. Additionally, the implementing system mayhave a configuration similar to system configuration 200.

In this example, process 300 is working with three sites (e.g., site 1,site 2, and site 3) that have copies of a disk, which may be a physicaldisk or a representation thereof. Each site also has multiple copies ofthe disk. A first disk at site 1 is the main copy of the disk, and afirst disk at site 2 is synchronously replicated from the first disk atsite 1 and serves as its primary backup. Site 3 provides an asynchronousbackup the disks at sites 1 and 2. Each site may, for example, be aseparate storage system in a separate location.

Process 300 calls for starting a synchronous replication between thefirst disk at site 1 and the first disk at site 2 (operation 304). Thisreplication may generally run throughout process 300. But there areinstances (e.g., when one of the disks is unavailable) when thisreplication will not be running. Process 300 then calls for waiting forthe initial copy between these disks to complete (operation 308).

Once the initial copy is complete, process 300 calls for triggeringpoint-in-time copies from the first disk at site 1 to a second disk atsite 1 and from the first disk at site 2 to a second disk at site 2 withrespect to host system input/output (TO) (operation 312). This mayinvolve quiescing host system IO. Process 300 also calls for starting asynchronous replication from the second disk at site 1 to a first diskat site 3 (operation 316) and checking if the replication between thedisks is complete (operation 320). If the replication is not complete,process 300 calls for waiting for the replication to complete.

Once the synchronous replication between the second disk at site 1 andthe first at site 3 has finished its initial copy, process 300 calls forstopping the replication (operation 324). Process 300 then calls fortriggering point-in-time copies from the first disk at site 3 to asecond disk at site 3 and from the first disk at site 3 to a third diskat site 3 (operation 328). The relative timing of these copies istypically unimportant. After this, consistent images exist on the firstdisk at site 1, the first disk at site 2, and the second disk at site 3,although the last will probably have an older image than the others.\

Process 300 also calls for cleaning the replication between the seconddisk at site 2 and the third disk at site 3 (operation 332). Asynchronous replication from the second disk at site 2 to the third diskat site 3 would be dirty, which means that the replication thinks itneeds to copy lots of data. However, all the work has been performed byoperations 316-328. Thus, the synchronization only needs to be cleaned.This may, for example, be accomplished by deleting the synchronousreplication and recreating it clean, which is commonly referred to aNOCOPY creation.

Process 300 then calls for retriggering the point-in time copies fromthe first disk at site 1 to the second disk at site 1 and from the firstdisk at site 2 to the second disk at site 2 (operation 336). Thisdirties the replications from the second disk at site 1 to the firstdisk at site 3 and from the second disk at site 2 to the third disk atsite 3 on the regions that have been updated by retriggering thepoint-in-time copies. Process 300 then calls for restarting thesynchronous replication from the second disk at site 1 to the first diskat site 3 (operation 316).

The cycle from operation 316 to operation 336 can run as fast as thesynchronous replication can complete copying the changes and the othercommands can be run. As the length of time that the replication copyingtakes is proportional to the number of writes written to the first diskat site 1 in the previous cycle, while the rate of writes written isless than the replication copying rate, the cycle period (and hence theRPO) can tend towards zero. And while the rate of writes is greater thanthe replication copying rate, the cycle period and the RPO will grow.This allows the system to cope with significant spikes in write rates.

This cycle generally runs as long as the sites are operating normally.Thus, the cycle can be repeated a large number of times during normaloperations. When one or the sites becomes unavailable, however, one ormore operations of the cycle may be stopped for a period of time (e.g.,until the lost site returns).

FIGS. 4A-D illustrate another example process 400 for managing theremote replication of data. Process 400 may, for example, be implementedin conjunction with process 300, which may be paused while parts ofprocess 400 are running.

Process 400 calls for determining whether site 1 is unavailable(operation 404). Site 1 may, for example, be lost due to a powerfailure, a communication failure, or a hardware failure. If site 1 hasnot been lost, process 400 calls for continuing to check whether site 1has been lost.

Once site 1 is unavailable, process 400 calls for failing host system IOover to site 2 (operation 408). This may, for example, be accomplishedusing the replication from the first disk at site 1 to the first disk atsite 2, which ensures that at the point at which site 1 is unavailable,the first disk at site 2 already contains the same data as the firstdisk at site 1. Thus, the host system IO may be failed overtransparently by a high-availability solution. Thus, host system IOsystem is now being written to the first disk at site 2.

Process 400 also calls for determining if operations 316-320 (i.e.,replicating the data from the second disk at site 1 to the first disk atsite 2) were being performed at the loss of site 1 (operation 412). Ifoperations 316-320 were being performed, the third disk at site 3 willcontain a consistent image of what the first disk at site 3 was at thebeginning of the cycle.

If a replication from the second disk at site 1 to the first disk atsite 2 was not being performed, process 400 calls for determiningwhether a point-in-time copy from the first disk at site 3 was beingperformed at the loss of site 1 (operation 416). If a point-in-time copyfrom the first disk at site 3 was not being performed at the loss,process 400 calls for determining whether a cleaning of the replicationfrom the second disk at site 2 to the third disk at site 3 was beingperformed at the loss of site 1 (operation 420).

If a cleaning of the replication from the second disk at site 2 to thethird disk at site 3 was not being performed, process 400 calls fortriggering a point-in-time copy from the first disk at site 2 to thesecond disk at site 2 (operation 424). Process 400 also calls startingthe synchronous replication from the second disk at site 2 to the thirddisk at site 3 (operation 428). Process 400 additionally calls fordetermining whether the synchronous replication from the second disk atsite 2 to the third disk at site 3 has finished its copy of changed data(operation 432). Once this synchronous replication has finished its copyof changed data, process 400 calls for stopping the replication(operation 436).

Process 400 also calls for triggering the point-in-time copy from thethird disk at site 3 to the disk second disk at site 3 (operation 440).Process 400 continues with determine whether site 1 is available(operation 444). If site 1 is not available, process 400 calls forreturning to operation 424. The cycle of operations 424-444 may berepeated until site 1 returns. The difference recording in the stoppedreplication from the first disk 1 at site 2 to the first disk at site 1while the process cycles through operations 424-444 allows site 1 to beincrementally resynchronized when it returns.

Returning to operation 412, if a replication from the second disk atsite 1 to the first disk at site 2 was being performed at the loss ofsite 1, process 400 calls for starting the replication from the seconddisk at site 2 to the third disk at site 3 (operation 428). Process 400may then continue from operation 428 as described above.

Returning to operation 416, if a point-in-time copy from the first diskat site 3 was being performed at the loss of site 1, process 400 callsfor cleaning the replication between the second disk at site 2 and thethird disk at site 3 (operation 448). Process 400 may then continue fromoperation 440.

Returning to operation 420, if a cleaning of the replication between thesecond disk at site 2 and the third disk at site 3 was being performedat the loss of site 1, process 400 calls for cleaning the replicationbetween the second disk at site 2 and the third disk at site 3(operation 452). Process 400 may then continue from operation 424.

When site 1 becomes available, process 400 calls for determining whethersite 2 is available (operation 456). If site 2 is available, process 400calls for restarting the synchronous replication between the first diskat site 2 and the first disk at site 1, although in the reversedirection from before (operation 460). Process 400 also calls fordetermining whether the incremental resynchronization between thesedisks is complete (operation 464). If the incremental resynchronizationis not complete, process 400 calls for waiting for it to complete.

Once the incremental resynchronization of the synchronous replicationfrom the first disk at site 2 to the first disk at site 1 is complete,process 400 calls for determining whether the current copy from thesecond disk of site 2 to the third disk of site 3 is complete (operation468). If the current copy from the second disk of site 2 to the thirddisk of site 3 is not complete, process 400 calls for waiting for it tocomplete.

Once the current copy from the second disk of site 2 to the third diskof site 3 is complete, process 400 calls for triggering thepoint-in-time copy from the third disk at site 3 to the second disk atsite 3 (operation 472) and triggering the point-in-time copy from thethird disk at site 3 to the first disk at site 3 (the reverse direction)(operation 476). The replication between the first disk at site 3 andthe second disk at site 3 may then be cleaned (operation 480). Thisreplication may be dirty, which means that the replication thinks itneeds to copy lots of data. However, all the work has been performed byoperations 468-476. Thus, only the replication needs to be cleaned. Thismay, for example, be accomplished by deleting the replication andrecreating it clean.

Process 400 is then at an end. Process 300 may, for example, then berestarted from operation 336. Additionally, the host system IO may againbe received on disk the first disk of site 1 by stopping the first diskat site 2 to the first disk at site 1 replication and restarting it inthe other direction. However, this is not required.

Returning to operation 456, if site 2 is not available, process 400calls for resynchronizing site 1 from site 3 (operation 484). Forexample, since the first disk of site 3 remains static while site 1 isunavailable, the differences recorded by the point-in-time copy from thefirst disk at site 3 to the second disk at site 3 allows incrementallyresynchronizing site 1 from site 3. Process 400 is then at an end.Process 300 may, for example, then be restarted from operation 336.Additionally, the host system IO may again be received on the first diskof site 1.

FIGS. 5A-B illustrate another example process 500 for managing theremote replication of data. Process 500 may, for example, be implementedin conjunction with process 300.

Process 500 calls for determining whether site 2 is unavailable(operation 504). Site 2 may, for example, be lost due to a powerfailure, a communication failure, or a hardware failure. If site 2 hasnot been lost, process 500 calls for continuing to check whether site 2has been lost.

Once site 2 unavailable, process 500 calls for ceasing to trigger thepoint-in-time copy from the first disk at site 3 to the third disk atsite 3 (operation 508). Thus, the host system IO may continue beingwritten to the first disk at site 1, and the main cycle of process 300(i.e., operations 316-336) may continue, except that nothing on site 2may be manipulated.

Process 500 then calls for determining whether site 2 is available(operation 512). If site 2 is not available, process 500 calls forwaiting for it to become available. During this time, the differencerecording in the stopped replication from the first disk of site 1 tothe first disk of site 2 allows site 2 to be incrementallyresynchronized when it returns.

When site 2 is available, process 500 calls for determining whether site1 is available (operation 516). If site 1 is available, process 500calls for restarting the synchronous replication from the first disk ofsite 1 to the first disk of site 2 (operation 520). Process 500 alsocalls for determining whether the resynchronization between these disksis complete (operation 524). If the resynchronization is not complete,process 500 waits for the resynchronization to complete.

Once the resynchronization is complete, process 500 calls fordetermining whether the replication from the second disk at site 1 tothe first disk at site 3 is complete (operation 528). If thisreplication is not complete, process 500 waits for the replication tocomplete. Once the replication is complete, process 500 calls forrestarting the trigger of the point-in-time copy from the first disk atsite 3 to the third disk at site 3 (operation 532). Process 500 is thenat an end, and process 300 can resume its normal operations beginning atoperation 328.

If, however, site 1 is not available, process 500 calls forresynchronizing site 2 from site 3 (operation 536). For example, sincethe third disk at site 3 remains static, the differences recorded by thepoint-in-time copy from this disk to the second disk at site 3 allowincremental resynchronization of site 2 from site 3. Process 500 is thenat an end, and process 300 can resume its normal operations beginning atoperation 328.

FIG. 6 illustrates another example process 600 for managing the remotereplication of data. Process 600 may, for example, be implemented inconjunction with process 300.

Process 600 calls for determining whether site 3 is unavailable(operation 604). Site 3 may, for example, be lost due to a powerfailure, a communication failure, or a hardware failure. If site 3 isnot unavailable, process 600 calls for continuing to check whether site3 is unavailable.

If site 3 becomes unavailable, process 600 calls for pausing replicationoperations except for the synchronous replication from the first disk atsite 1 to the first disk at site 2 (e.g., operations 316-336) (operation608). Process 600 also calls for determining whether site 3 has becomeavailable (operation 612). If site 3 is not available, process 600 callsfor waiting for it to become available.

When site 3 is available, process 600 calls for determining whether site1 is available (operation 612). If site 1 is available, process 600calls for resuming replication operations from the paused point(operation 616). Process 600 is then at an end, and process 300 mayresume operation.

If, however, site 1 is not available, process 600 calls forresynchronizing site 3 from site 2 (operation 620). This may, forexample, be accomplished use operations 428-440 from process 400.Process 600 is then at an end, and process 300 may resume operation.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of systems, methods, andcomputer program products of various implementations of the disclosure.In this regard, each block in the flowchart or block diagrams mayrepresent a module, segment, or portion of code, which can include oneor more executable instructions for implementing the specified logicalfunction(s). It should also be noted that, in some alterativeimplementations, the functions noted in the blocks may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or the flowchart illustration, and combination ofblocks in the block diagrams and/or flowchart illustration, can beimplemented by special purpose hardware-based systems the perform thespecified function or acts, or combinations of special purpose hardwareand computer instructions.

FIG. 7 illustrates an example computer system 700 for managing theremote replication of data. Computer system 700 may, for example,illustrate some of the components of a storage controller of a storagesystem.

System 700 includes a processor 710, an input/output system 720, andmemory 730, which are coupled together by a network 740. As illustrated,computer system 700 is functioning as a storage controller of a storagesystem.

Processor 710 typically includes a logical processing unit (e.g., anarithmetic logic unit) that processes data under the direction ofprogram instructions (e.g., from software). For example, processor 710may be a microprocessor, a microcontroller, or an application specificintegrated circuit. The processor may operate by reduced instruction setcomputer (RISC) or complex instruction set computer (CISC) principles.In general, the processor may be any device that manipulates data in alogical manner.

Input/output system 720 may include one or more communication interfacesand/or one or more other user interfaces. A communication interface may,for instance, be a network interface card (whether wireless or wireless)or a modem. A user interface could, for instance, be a user input device(e.g., a keyboard, a keypad, a touchpad, a stylus, or a microphone) or auser output device (e.g., a monitor, a display, or a speaker). Ingeneral, system 720 may be any combination of devices by which acomputer system can receive and output data.

Memory 730 may, for example, include random access memory (RAM),read-only memory (ROM), flash memory, and/or disc memory. Various itemsmay be stored in different portions of the memory at various times.Memory 730, in general, may be any combination of devices for storingdata.

Memory 730 includes instructions 732 and data 736. Instructions 732include an operating system 733 (e.g., Windows, Linux, or Unix) andapplications 734, which include a replication management program 735.Data 736 includes the data required for and/or produced by applications734, including site data 737, disk data 738, and write data 739.

Network 740 is responsible for communicating data between processor 710,input/output system 720, and memory 730. Network 740 may, for example,include a number of different types of busses (e.g., serial andparallel).

In certain modes of operation, processor 710 is able to receive writes(e.g., from host system IO or from another storage system) and preparethem for storage managed by the computer system. Once stored, processor710 may report their storage (e.g., to the host system IO or anotherstorage system). Processor 710 can also trigger replications (e.g.,synchronous or point-in-time copies) between various disks, which may bein its managed storage or in the managed storage of another computersystem. The disks may be actual disks or representations of disks.Furthermore, processor 710 can determine if various replications need tobe paused (e.g., if another site is no longer available). Moreover,processor 710 can determine how to resynchronize various disk (e.g., ifanother site becomes available). Processor 710 may, for example,accomplish this by implementing one or more parts of processes 300-600.

The terminology used herein is for the purpose of describing particularimplementations only and is not intended to be limiting. As used herein,the singular form “a”, “an”, and “the” are intended to include theplural forms as well, unless the context clearly indicates otherwise. Itwill be further understood that the terms “comprises” and/or“comprising,” when used in the this specification, specify the presenceof stated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups therefore.

The corresponding structure, materials, acts, and equivalents of allmeans or steps plus function elements in the claims below are intendedto include any structure, material, or act for performing the functionin combination with other claimed elements as specifically claimed. Thedescription of the present implementations has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the implementations in the form disclosed. Manymodification and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of thedisclosure. The implementations were chosen and described in order toexplain the principles of the disclosure and the practical applicationand to enable others or ordinary skill in the art to understand thedisclosure for various implementations with various modifications as aresuited to the particular use contemplated.

A number of implementations have been described for managing the remotereplication of data, and several others have been mentioned orsuggested. Moreover, those skilled in the art will readily recognizethat a variety of additions, deletions, modifications, and substitutionsmay be made to these implementations while still achieving the remotereplication of data. Thus, the scope of the protected subject mattershould be judged based on the following claims, which may capture one ormore concepts of one or more implementations.

The invention claimed is:
 1. A system comprising: a first site having afirst disk for storing input/output data for a host system, the firstsite further having a second disk; a second site that is remote from thefirst site, the second site having a first disk and a second disk; and athird site that is remote from the first site and the second site, thethird site having a first disk, a second disk, and a third disk; and aprocessor configured to: initiate a first synchronous replication of thedata of the first disk of the first site to the first disk of the secondsite; responsive to a completion of a copy of the first synchronousreplication: trigger a point-in-time copy of data of the first disk ofthe second site to the second disk of the second site; and trigger apoint-in-time copy of the data of the first disk of the first site tothe second disk of the first site; and initiate a second synchronousreplication of the data of the second disk of the first site to thefirst disk of the third site and, upon completion of a copy of thesecond synchronous replication, stop the second synchronous replication.2. The system of claim 1, wherein the processor is configured to triggera point-in-time copy from the first disk at the third site to the seconddisk at the third site and trigger a point-in-time copy from the firstdisk at the third site to the third disk at the third site.
 3. Thesystem of claim 2, wherein the processor is configured to: determinewhether the second site is unavailable and cease, if the second site isunavailable, triggering the point-in-time copy from the first disk atthe third site to the third disk at the third site; determine whetherthe second site is available and restart, if the second site isavailable, the first synchronous replication from the first disk at thefirst site to the second disk at the second site, determine whetherresynchronization of the first disk at the first site to the first diskat the second site has occurred, determine, if the resynchronization ofthe first disk at the first site to the first disk at the second sitehas occurred, whether the replication from the second disk at the firstsite to the first disk at the third site is complete; and restart thetriggering of the point-in-time copy from the first disk at the thirdsite to the third disk at the third site if the replication from thesecond disk at the first site to the first disk at the third site iscomplete.
 4. The system of claim 2, wherein the processor is configuredto determine whether the third site is unavailable, pause, if the thirdsite is unavailable, the replication operations except for thereplication from the first disk at the first site to the first disk atthe second site, determine whether the third site is available, andrestart, if the third site is available, the replication operations fromthe paused point.
 5. The system of claim 1, wherein the processor isconfigured to determine whether the first site is unavailable andswitch, if the first site is unavailable, host system input/output overto the first disk at the second site.
 6. The system of claim 5, whereinthe processor is configured to: trigger a point-in-time copy of thefirst disk at the second site to the second disk at the second site andto start a third synchronous replication of the second disk at thesecond site to the third disk at the third site; and trigger apoint-in-time copy of the third disk at the third site to the seconddisk at the third site.
 7. The system of claim 5, wherein the processoris configured to: determine whether the first site is available andstart, if the first site is available, a third synchronous replicationfrom the first disk at the second site to the first disk at the firstsite, and determine whether a current copy from the second disk at thesecond site to the third disk at the third site is complete; andtrigger, if the copy from the second disk at the second site to thethird disk at the third site is complete, a point-in-time copy from thethird disk at the third site to the second disk at the third site andtrigger a point-in-time copy from the third disk at the third site tothe first disk at the third site.
 8. A system comprising: a first sitestoring a first disk for input/output for a host system; a second sitethat is remote from the first site, the second site storing a first diskthat is a synchronous replication of the first disk at the first site;and a third site that is remote from the first site and the second site,the third site storing a first disk that is an asynchronous replicationof the first disk at the first site; wherein the first site stores asecond disk that is a point-in-time copy of the first disk of the firstsite; wherein the third site stores a first disk, a second disk, and athird disk, wherein the first disk at the third site is an intermittentsynchronous replication of the second disk at the first site, the seconddisk at the third site is a point-in-time copy of the first disk at thethird site, and the third disk at the third site is a point-in-time copyof the first disk at the third site; and wherein first site isconfigured to trigger a point-in-time copy from the first disk at thefirst site to the second disk at the first site, start a synchronousreplication from the second disk at the first site to the first disk atthe second site, and stop the synchronous replication when the copy iscomplete.
 9. The system of claim 8, wherein: the second site isconfigured to trigger a point-in-time copy from the first disk at thesecond site to a second disk at the second site; and the third site isconfigured to trigger a point-in-time copy from the first disk at thethird site to the second disk at the third site and trigger apoint-in-time copy from the first disk at the third site to the thirddisk at the third site.
 10. The system of claim 9, wherein: the thirdsite is configured to determine whether the second site is unavailableand cease, if the second site is unavailable, triggering thepoint-in-time copy from the first disk at the third site to the thirddisk at the third site; the first site is configured to determinewhether the second site is available, restart, if the second site isavailable, a synchronous replication from the first disk at the firstsite to the second disk at the second site, determine whetherresynchronization of the first disk at the first site to the first diskat the second site has occurred, determine, if the resynchronization ofthe first disk at the first site to the first disk at the second sitehas occurred, whether the replication from the second disk at the firstsite to the first disk at the third site is complete; and the third siteis configured to restart the triggering of the point-in-time copy fromthe first disk at the third site to the third disk at the third site ifthe replication from the second disk at the first site to the first diskat the third site is complete.
 11. The system of claim 9, wherein thefirst site is configured to determine whether the third site isunavailable, pause, if the third site is unavailable, the replicationoperations except for the replication from the first disk at the firstsite to the first disk at the second site, determine whether the thirdsite is available, and restart, if the third site is available, thereplication operations from the paused point.
 12. The system of claim 8,wherein the second site is configured to determine whether the firstsite is unavailable and switch, if the first site is unavailable, hostsystem input/output over to the first disk at the second site.
 13. Thesystem of claim 12, wherein: the second site is configured to trigger apoint-in-time copy of the first disk at the second site to the seconddisk at the second site and to start a synchronous replication of thesecond disk at the second site to the third disk at the third site; andthe third site is configured to trigger a point-in-time copy of thethird disk at the third site to the second disk at the third site. 14.The system of claim 12, wherein: the second site is configured todetermine whether the first site is available and start, if the firstsite is available, a synchronous replication from the first disk at thesecond site to the first disk at the first site, and determine whether acurrent copy from the second disk at the second site to the third diskat the third site is complete; and the third site is configured totrigger, if the copy from the second disk at the second site to thethird disk at the third site is complete, a point-in-time copy from thethird disk at the third site to the second disk at the third site andtrigger a point-in-time copy from the third disk at the third site tothe first disk at the third site.